feat(rag): hybrid docs retrieval and readable ingest path - #249

Three test modules abort at collection on main:

    ImportError: cannot import name 'clean_content' from 'utils'
    (unknown location)

`pytest` exits 2, so the whole suite stops before running and both
workflows that invoke it are red.

kubeflow#249 moved `utils.py` into `pipelines/utils/`. These modules put only
`pipelines/` on `sys.path`, where the sibling directory of the same name
now shadows the module as a namespace package — `utils.__file__` is None
and `utils.__path__` is the directory, so no attribute resolves.

The pipelines themselves already handle this with a `_UTILS_DIR` entry
pointing at `pipelines/utils`; this applies the same thing to the tests
that import `utils` directly.

Recovers 72 tests across the four affected modules. The 7 failures and 63
errors that remain are pre-existing on dedbc84 and are not touched here:
`test_mcp_server.py` references `server.client`, removed when kubeflow#249 split
the MCP server, and `test_docs_pipeline.py` exercises the `store_milvus`
component that kubeflow#249 replaced with `utils/milvus_store.py`.

Signed-off-by: Abhishek <abhikokadwar2@gmail.com>

Every check on main has failed since #249/#261:

- Pytest: the pipeline tests imported code_utils, issues_utils and utils
  from pipelines/, but #249 moved them to pipelines/utils/. Add that
  directory to sys.path, as the pipelines themselves do.
- Widget tests: #249 intentionally stopped rendering links in the answer
  bubble (sources go to the Sources panel). Update the three tests to
  that contract: http(s) links render as their escaped label, other
  links stay inert text, inline code is untouched.
- test_mcp_server.py targets pre-#249 internals (server.client, _init,
  _rerank_hits, ...) that no longer exist. Skip the module with the
  reason; it needs a rewrite against milvus_search/intent_router.
- Compile: kubeflow-pipeline.py needs the ingest libraries
  (python-frontmatter, html-table-rescuer, ...). Compile the pipelines
  in their own virtualenv from pipelines/requirements.txt.
- pipelines/requirements.txt could not be installed at all: kfp 2.17.0
  pins requests==2.33.0 but the file pinned 2.34.2.
- ruff format on milvus_search.py and server.py.

Locally, with requirements-test.txt alone and with the MCP server
requirements added: 112 passed, 69 skipped. All three pipelines compile
in a clean virtualenv. ruff check and format pass.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

utils/hugo_ingest.py had no tests. kubeflow#249's test plan names
tests/test_hugo_ingest.py, but the file was never committed (kubeflow#259).

Only the production surface is covered. canonical_rag_ingest imports
parse_frontmatter and process_html_table; clean_hugo_markdown has no
caller, so testing it would add coverage that exercises nothing.

Expectations are taken from the module's docstrings and inline comments,
from how parse_canonical_document consumes the results, and from
docs/RAG_V4_ARCHITECTURE.md, rather than from whatever the code currently
returns:

- parse_frontmatter: YAML and TOML give the title/description/weight that
  parse_canonical_document reads; a byte-order mark does not hide the
  frontmatter; missing keys stay absent; empty input is returned as-is;
  malformed frontmatter falls back to ({}, content) instead of aborting
  the run.
- process_html_table: tables become GFM tables; rowspan values repeat
  into every spanned row (release tables); colspan keeps rows the same
  width; an escaped pipe does not add a column; text around and between
  tables stays off the table rows, as canonical_rag_ingest groups rows;
  the markdownify fallback still produces a table; text with no table is
  returned unchanged.

The ingest dependencies were only in pipelines/requirements.txt and
Dockerfile.pipeline, so the tests would have skipped in PR Safety. They
are added to requirements-test.txt at the same pinned versions.

Signed-off-by: Abhishek <abhikokadwar2@gmail.com>